Skip to content

feat(sync): add sync_include_cwd_prefixes ingestion filter - #1004

Merged
wesm merged 7 commits into
kenn-io:mainfrom
RobSchilderr:feat/sync-include-cwd-prefixes
Jul 7, 2026
Merged

feat(sync): add sync_include_cwd_prefixes ingestion filter#1004
wesm merged 7 commits into
kenn-io:mainfrom
RobSchilderr:feat/sync-include-cwd-prefixes

Conversation

@RobSchilderr

@RobSchilderr RobSchilderr commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Adds an opt-in sync_include_cwd_prefixes config option that restricts local session ingestion to sessions whose working directory lives under one of the configured path prefixes.

Motivation: on a machine shared across multiple clients or workspaces, the archive currently indexes every session it can find, so transcripts from unrelated (possibly confidential) workspaces end up side by side in one database and UI. The DuckDB/PG mirror projects filters can carve out a scoped read-only view, but that requires a second serving process and an exact project-name list that goes stale as projects are added. A cwd allow-list at ingestion solves this at the source, works uniformly across agents (including agents like Codex whose session files are date-organized rather than per-project), and keeps everything in the normal serve flow.

sync_include_cwd_prefixes = [
  "/home/me/work/client-a",
  "/home/me/oss",
]

What changed

  • internal/sync/cwd_filter.go (new): cwdPrefixFilter with normalization (trim, drop blanks, strip trailing separators) and path-boundary-aware matching (/a/b matches /a/b/c but not /a/bc; both / and \ boundaries).
  • internal/sync/engine.go: EngineConfig.IncludeCwdPrefixes; the filter is enforced as a veto in prepareSessionWrite, before the preserve/merge handling, so every full-write path across all agents is covered by the existing central seam. Callers already treat the veto as an intentional skip.
  • internal/config/config.go: SyncIncludeCwdPrefixes (toml:"sync_include_cwd_prefixes", config-file only, not exposed via the settings API).
  • cmd/agentsview/*.go, internal/server/huma_routes_sync.go: the option is wired into every local engine construction site. The remote-sync engine (internal/remotesync) is deliberately left unwired — the prefixes describe local paths, so applying them to remote hosts' cwds would silently drop remote sessions.
  • cmd/agentsview/cli.go: --help section documenting the option.
  • docs/configuration.md: new "Restricting Ingestion by Working Directory" section under Sync Behavior.
  • Tests: unit table for the matcher (cwd_filter_test.go), config load tests, and an engine integration test (cwd_filter_integration_test.go) syncing one in-prefix, one out-of-prefix, and one boundary-sibling (/Users/alice/workspace vs prefix /Users/alice/work) Claude session.

Semantics / decisions

  • Empty list (default): no behavior change.
  • Sessions without a recorded cwd are skipped while the filter is set — they can't be attributed to a workspace, and an allow-list that silently admits unattributable sessions wouldn't be an allow-list. Documented in the config docs and --help.
  • The filter gates ingestion only. Sessions already in the archive are preserved, consistent with the persistent-archive policy in AGENTS.md; the docs point at agentsview prune for cleaning up previously ingested sessions.
  • Matching is case-sensitive and ~ is not expanded (documented).
  • The incremental append path is unaffected by design: it only appends to sessions that already exist in the archive, which is exactly the pre-existing-data behavior above.

Where to look

The veto placement in prepareSessionWrite is the main review surface — it sits after the worktree-project resolution and before shouldPreserveOpenCodeFormatArchive, so a filtered session is never written by any downstream path. The other question worth a look is the empty-cwd decision; happy to flip it to allow-when-missing if you'd rather bias toward completeness.

@RobSchilderr
feat(sync): add sync_include_cwd_prefixes ingestion filter
b6c65f3 
Add an opt-in config option that restricts local session ingestion to
sessions whose working directory lives under one of the configured
path prefixes. The filter is enforced as a veto in prepareSessionWrite
so every full-write path across all agents is covered uniformly.
Sessions without a recorded cwd are skipped while the filter is set;
remote-host sync is unaffected because the prefixes describe local
paths. The filter gates ingestion only: sessions already in the
archive are preserved, consistent with the persistent-archive policy.
@roborev-ci

roborev-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown

roborev: Combined Review (b6c65f3)

Medium finding blocks clean approval.

Medium

  • internal/sync/engine.go:4852, internal/sync/engine.go:6903
    The new sync_include_cwd_prefixes filter is enforced in prepareSessionWrite, but append-only incremental sync bypasses that path. Existing archived sessions with a stored cwd outside the allow-list can still receive appended messages through tryIncrementalJSONL / writeIncremental, making excluded content visible through the archive/API and potentially downstream sync.
    Fix: When the cwd filter is active, reject incremental updates whose inc.cwd / inc.Cwd is not allowed, or force those updates through the full parse path so prepareSessionWrite applies the same veto. Add a regression test for an existing outside-prefix session receiving appended messages.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 5m2s

@RobSchilderr
fix(sync): enforce cwd allow-list on the incremental append path
368b2c7 
tryIncrementalJSONL now falls back to a full parse for sessions whose stored cwd is outside the allow-list, so prepareSessionWrite applies the same veto, and writeIncremental refuses filtered updates as a seam guard for any future producer. Adds a regression test for an archived outside-prefix session receiving appended messages.
@RobSchilderr

RobSchilderr commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the roborev medium finding in 368b2c7:

  • tryIncrementalJSONL now vetoes sessions whose stored cwd is outside the allow-list right after the incremental lookup, falling back to the full parse path so prepareSessionWrite applies the same veto (this also re-derives the cwd from the whole file, covering rows that predate cwd capture).
  • writeIncremental refuses filtered updates as a defense-in-depth guard at the write seam, so no future producer can bypass the filter.
  • Regression test TestSyncEngineCwdPrefixFilterBlocksIncrementalAppend: archives an outside-prefix session unfiltered, enables the filter, appends to the file, and asserts no new messages land. Verified red without the engine change, green with it.

@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (368b2c7)

Medium severity issue found; security review reported no additional issues.

Medium

  • internal/sync/engine.go:5807 - Filtered sessions are returned as a generic “not written” case, so a full ResyncAll where every discovered session is outside sync_include_cwd_prefixes ends with Synced == 0, Failed == 0, and no parser-excluded marker. The resync abort guard then treats the run as an unsafe empty rebuild and aborts, leaving NeedsResync() true on every startup even though this is an intentional all-filtered result.
    • Fix: Track cwd-filtered sessions distinctly in sync stats and let the resync path treat an all-filtered run as intentional so it can proceed to orphan-copy preserved archive rows; add a resync test where the allow-list excludes all existing sessions.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 11m28s

@wesm

wesm commented Jul 7, 2026

Copy link
Copy Markdown
Member

looking

@wesm
fix(sync): treat an all-cwd-filtered resync as intentional
eef4cd7 
A full resync where sync_include_cwd_prefixes vetoes every discovered
session previously ended with Synced == 0, Failed == 0, and no
parser-excluded marker, so the resync abort guard read the run as an
unsafe empty rebuild, aborted the swap, and left NeedsResync true on
every startup.

prepareSessionWrite now returns a verdict that distinguishes the cwd
allow-list veto from archive-preserve vetoes, the write batch paths
count filtered sessions into sync stats, and the abort guard treats an
all-filtered run as intentional so the swap proceeds and the orphan
copy restores the archived rows.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (eef4cd7)

No Medium, High, or Critical findings were reported.

Both reviews are clean at the requested severity threshold.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 8m31s

wesm added 2 commits July 7, 2026 15:51
@wesm
Merge remote-tracking branch 'origin/main' into robschilderr/feat/syn…
8995e92 
…c-include-cwd-prefixes

* origin/main:
  feat(usage): show session context and token breakdown (kenn-io#982) (kenn-io#989)
  perf(push): ignore volatile stat fields for session candidacy (kenn-io#1014)
  feat(settings): support worktree layout mappings (kenn-io#582) (kenn-io#993)
  fix(config): apply port from config.toml to Config struct (kenn-io#1005)
  feat(parser): add Qoder session support (kenn-io#1013)
  fix(usage): wire agent exclusions through usage filters (kenn-io#972)
  fix(frontend): preserve calendar range picker selections (kenn-io#1016)
  feat: semantic search with run-grouped embeddings and conversation-unit citations (kenn-io#999)
  feat(parser): add ZCode SQLite sync support (kenn-io#1003) (kenn-io#1012)
  fix(sync): drop unchanged opencode-family container sessions (kenn-io#1015)
  fix(sync): skip local git discovery for foreign-machine sessions (kenn-io#1008)
  fix(activity): count subagent sessions in activity report cost (kenn-io#1006)
  feat(i18n): add Korean (ko) locale support (kenn-io#1002)
@wesm
fix(sync): update Qoder test for four-value writeBatch
8302b5e 
Merging main brought in the Qoder parser, whose test helper still
called writeBatch with three return values. Discard the new
cwd-filtered count like the other test call sites.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (8302b5e)

Medium severity finding:

  • internal/sync/cwd_filter.go:48: allows() compares raw cwd strings and treats both / and \ as path boundaries. Paths such as /home/me/work/../client-b, or POSIX paths containing backslashes like /home/me/work\client-b, can pass an allow-list of /home/me/work even when they are outside that directory. This can bypass the sync_include_cwd_prefixes privacy boundary and expose excluded transcripts through archive/search/UI.

Suggested fix: normalize prefixes and cwd using local filesystem semantics before matching. Use filepath.Clean, preserve filesystem roots and drive roots, require absolute paths, and prefer filepath.Rel or os.PathSeparator checks. Do not treat \ as a separator on POSIX.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 7m24s

@wesm
fix(sync): match cwd prefixes with local filesystem semantics
db63203 
The cwd allow-list compared raw strings and treated both slash and
backslash as path boundaries on every platform. A recorded cwd such as
/home/me/work/../client-b, or a POSIX directory literally named
"work\client-b", could match a /home/me/work prefix while living
outside it.

Clean prefixes and cwds with filepath.Clean before matching and use
os.PathSeparator as the only boundary, so .. components resolve
lexically and a backslash is an ordinary filename character on POSIX.
A root prefix now matches instead of being silently dropped. Document
the matching semantics in docs/configuration.md.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (db63203)

High-level verdict: the PR has one high-risk data-loss issue and one medium resync-guard correctness issue to address.

High

  • internal/sync/engine.go:3317
    Parser-excluded or stale session IDs can be deleted before the new cwd filter veto runs in prepareSessionWrite. For an outside-prefix source that emits excludedSessionIDs or changes IDs, the old archived row can be hard-deleted or excluded from resync orphan copy, then the replacement is skipped by the cwd filter.
    Fix: Gate parser-exclusion deletes and stale file-path cleanup by the cwd allow-list, or defer them until after determining that the source/result is allowed to write.

Medium

  • internal/sync/engine.go:1267
    cwdFilteredOnly is true when any session was cwd-filtered, not when the zero-synced run was fully explained by cwd filtering. This can suppress the existing resync abort guard for mixed zero-write runs where other discovered sources were skipped or dropped for unrelated reasons.
    Fix: Require all zero-write sessions/files to be accounted for by cwd filtering and other intentional buckets, using a file/session counter rather than cwdFilteredSessions > 0.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 8m7s

@wesm
fix(sync): freeze parser exclusions and tighten resync guard under cw…
20c6026 
…d filter

Two review findings on the cwd allow-list:

Parser exclusions (including engine stale-row cleanup) ran before the
cwd veto, so a source outside the allow-list could hard-delete an
archived row while its replacement write was vetoed, and the excluded
ID also skipped resync's orphan copy. Both delete seams now require
the source to prove at least one allowed session or incremental
update; otherwise its exclusions are dropped and archived rows
survive, matching the documented ingestion-only contract.

The resync abort guard's cwd-filtered escape fired when any session
was filtered, which could excuse mixed zero-write runs. collectAndBatch
now drops vetoed sessions before batching and counts fully filtered
files, and the escape additionally requires every OK file to be
accounted for as cwd-filtered or parser-excluded. The guard moved into
shouldAbortResyncSwap so its branches are unit-testable.
@roborev-ci

roborev-ci Bot commented Jul 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (20c6026)

No issues found.

Reviewers: 2 done | Synthesis: codex | Total: 9m46s

@wesm

wesm commented Jul 7, 2026

Copy link
Copy Markdown
Member

Makes sense. Merging

@wesm
wesm merged commit 06778a5 into kenn-io:main Jul 7, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@RobSchilderr @wesm